In the field of hydrology, hydrologists and other environmental scientists apply scientific knowledge and mathematical principles to solve water-related problems such as quantity, quality and availability. They may be concerned with finding water supplies for cities or irrigated farms, or controlling river flooding or soil erosion. Or, they may work in environmental protection: preventing or cleaning up pollution or locating sites for safe disposal of hazardous wastes.
Much of an hydrologist or environmental scientists work relies on computers for organizing, summarizing and analyzing masses of data collected from rivers, water wells and weather stations, and for modeling studies such as the prediction of flooding and the consequences of reservoir releases or for example the effect of leaking underground oil storage tanks.
The data is collected in one of two ways, by manual field measurements or by aquatic monitoring sensors. The latter replacing the traditional manual approach which tends not to capture extreme events, such as storms or pollution spills unless samplers are unlikely to be in the field exactly when such events occur. Moreover, occasional field sampling cannot characterize higher-frequency aquatic processes, such as the diurnal oscillations of pH, dissolved oxygen, and conductivity that can result from biological activity or temperature.
On the other hand, aquatic monitoring sensors can often produce data that may not be representative of actual conditions. For example, optical (turbidity) sensors are prone to record unrealistically high values due to bubble disturbances, wiper brush positioning, or biological fouling of the sensor window. Sensors such as pH and dissolved oxygen can be miscalibrated, or if damaged can begin to drift as the control solution becomes contaminated with ambient water. Water level sensors can produce spurious data if the sensor float becomes jammed due to frazil ice or if pressure transducers are improperly calibrated, deployed, or subject to large temperature oscillations. Even solid-state sensors, such as thermistors, can record non representative values when exposed to air during low flow periods.
Additionally datasets that are produced by automated sensors are much larger than traditional manual grab sampling datasets. A weekly grab sampling program would produce 52 data points per year whereas a automated sensor recording every 15 minutes would collect nearly double that in a single day. Tools used to analyze traditional water quality data such as MS Excel are ill suited to the large datasets produced by data loggers. Excel is limited to plotted 32 thousand data points (1 year of 15 minute data), and cannot hold more than 64 thousand data points in a single column. Viewing more than a few years worth of data becomes impractical quickly. Furthermore validation and correction to time series datasets in tools such as Excel is hugely inefficient.
Recognising this problem a number of software tools have been produced to aid the hydrologist in the various tasks of organizing, summarizing, analyzing and validating masses of this data. This data can be time series data, discrete sample data or a combination. A number of these tools are freely available at the United States Geological Services (USGS) website http://water.usgs.gov/software/surface_water.html.
Hydrologists and environmental scientists face unique problems with respect to the data that they have to work on. The data collected is generally from disparate, physically isolated sensors and because of cost constraints there is little redundancy in the data collected to aid in verifying the validity of the data. Unlike in most engineering or scientific research fields, such as in a laboratory or process plant where the data sensors can be easily verified or re-measured, environmental scientists have to make do with data instead of just throwing it out or using data from redundant sensors.
Accordingly the environmental scientist and hydrologists must employ a myriad of different mathematical techniques for validating and correcting data. As a result a large assortment of different software tools are available data validity checking and correction. Often a number of different tools are required to be used in a variety of sequences or process steps. Again the USGS website provides a list of many such tools.
For example, data validation tools are used to estimate point-by point data uncertainty in time series data, since a series of data points over time (time series data) are only useful if they reflect true conditions, it is necessary to assess the reliability of the time series data. Data flagging tools are used for identification of questionable data. Data correction tools are used for removing outliers or non physical data values or to correct for fouling or sensor drift or interpolate sections of missing data.
Modeling tools may use rating curves, which express a relationship between stage and discharge at a cross section of a river. In most cases, data from stream gauges are collected as stage data. In order to model the streams and rivers, the data needs to be expressed as stream flow using rating tables. Conversely, the output from a hydrologic model is a flow, which can then be expressed as stage for dissemination to the public.
It is time consuming to arrange the order of processes, to include or exclude processes, and to ensure data compatibility and quality between steps. For example in a peak flow determination exercise if simulated peak flows differed from observed peak flows, this may be due to an error in the data and a new process step involving data correction may have to be included. A solution is proposed in U.S. Pat. No. 6,889,141 which teaches a method and system consisting of several independent automatic steps, wherein one step's result can be linked as the input of other steps through spreadsheets or text files to form a seamless stream of calculation. However this system is still limited in that it restricts the user to a rigid sequence of predetermined process steps and does not allow user the flexibility of graphically constructing and editing their own sequence of process steps.
Accordingly there is a need for an integrated hydrological system which offers to hydrologists and environmental scientists a convenient computer based environment for the entire hydrological data management process, starting with a data stream from telemetry equipment and data processing with the time series management, and reporting.
There is also a need for a system and method that significantly reduces the time required to organize, validate, correct, and plot hydrometric and water quality data.
There is also a need for a system and method that allows for easy statistical analysis, reporting, data grading, and modeling of this data.
The is a still further need for an environment that simplifies the management and analysis of water quality, hydrology, and climate time-series data by allowing users to define their entire data processing workflow in a highly intuitive and graphical workspace.